Journal of Vision

In fully-sighted individuals, pursuit and saccadic eye movements work together to foveate moving targets; pursuit attempts to match target velocity, while catch-up saccades re-align the fovea to the target if it deviates during pursuit. Although saccade-pursuit interactions are well understood in healthy individuals, much less is known about how these mechanisms adapt in the presence of central vision loss. Here, we simulated central vision loss in healthy young adults to examine plasticity in saccade-pursuit mechanisms and investigated whether and how catch-up saccades adapted over five days of visual tracking. Twenty-five fully sighted participants performed a task in which they pursued a moving target that stepped position randomly (designed to elicit catch-up saccades) while viewing with a gaze-contingent central scotoma. The scotoma was shaped like a bitten cookie to encourage use of a specific preferred retinal locus (PRL) corresponding to the peripheral location with the highest acuity. By the end of the five days of repetition, most participants showed catch-up saccade endpoints shifted towards the PRL rather than the fovea. On average, endpoints shifted 0.5 deg towards the PRL between days 1 and 5 (range=0.3-1.5 deg), with significant shifts in 13 of the 25 participants. Adaptation was asymmetric; shifts were largest when the target moved in the direction opposite the scotoma bite and smallest when it moved towards it. In summary, we observed adaptation of saccade-pursuit interactive mechanisms through the re-referencing of catch-up saccades from the fovea to PRL. These results provide a promising avenue for rehabilitation in central vision loss aimed at improving eye movements and encouraging PRL use. Ultimately, such an approach may enhance residual visual function and quality of life.

When objects are embedded in a complex lighting environment, the illumination they receive depends on their geometric relationship to the environment. We measured performance in an odd-one-out color constancy task, with objects rendered in complex lighting environments. Observers identified the odd object among four objects, with two rendered under each of two lighting environments, sampled from a wide range of object orientations and camera positions. Constancy was high. The presence of a surrounding context increased accuracy. Gloss provided no measurable benefit. We also tested a condition in which the global color impression provided misleading information about the lighting environment, but this impaired performance only slightly. These findings suggest that human color constancy remains robust even under complex lighting environments, contrary to predictions of many simple models of illuminant discounting.

Face recognition ability varies enormously across humans. Individuals with face-blindness (prosopagnosia) struggle to recognize even close family members while super-recognizers can identify strangers with exceptional accuracy. Where do ANNs fall within the human face recognition spectrum? Here, we administered the standardized tests used to characterize human face recognition ability to a diverse set of models, allowing us to contextualize a model's performance within the distribution of human behavior. We found that the majority 55% of models to be classified as face-blind and that even the best face-trained models do not cross the human threshold to be considered a super-recognizer. Interrogating the internal representations of these models showed that models that performed well on standardized face recognition tasks were more identity-selective and viewpoint-invariant. We also found that low-performing models did contain some identity information in independent representational subspaces. Removing viewpoint-dependent subspace improved face recognition abilities in 49 of the 53 models tested. Targeted unit ablations further identified opposing contributions, with viewpoint-dependent units disrupting identity coding and viewpoint-tolerant units supporting it. Together, our results show that most AI models are face-blind with worse face recognition ability than humans, and demonstrate how differences across AI models can be used generate testable hypotheses about the computational basis of human face recognition in humans which can then be probed in future studies.

Eye movements have been shown to reveal the cognitive processes underlying spatial ability, but most studies investigating gaze behaviour in spatial cognition use a single, screen-based task. Therefore, it is unknown whether these gaze behaviours are preserved across naturalistic task environments, or whether their relationship with spatial performance is domain-general or task-specific. The present study tested whether established gaze behaviours are preserved and generalizable across diverse, naturalistic spatial tasks. N = 30 young adults completed three naturalistic spatial tasks varying in perceptual, cognitive, and visuomotor demands: Q-bitz, the Brick Building Task (BBT), and the Mental Rotation Test (MRT). Gaze was measured using the Pupil Labs Neon eye tracker. Within-task analyses were conducted to assess whether gaze behaviours were preserved, and cross-task analyses were conducted to assess the generalizability of preserved gaze behaviours. Within-task analyses identified source inspection and transition rate as gaze behaviours that were consistently sensitive to changing task demands. Cross-task analyses showed that source fixation duration was a domain-general predictor of performance, whereas the relationship between transition rate and performance differed by task. Exploratory sex-separated analyses also suggested a more consistent gaze-performance relationship in males. These findings show that source fixation duration may represent a common cognitive process underlying performance across naturalistic spatial tasks.

Changes in pupil size have been widely used as an indirect measure of attention. However, attention engages on a faster timescale than the pupillary response. Therefore, the slow time course of the pupil response makes it difficult to distinguish attentional responses to stimuli presented in close temporal succession. Although signal deconvolution has been used in previous research to estimate latent attentional responses from the pupil signal, existing methods primarily model dilation without considering the visual stimulus evoked constriction phase of the pupil. Here, we introduce a dual-kernel deconvolution model that models constriction and dilation using separate impulse response functions. Using an RSVP paradigm with emotional faces as target stimuli, we demonstrate that the proposed model estimates latent attentional response despite substantial overlap in the measured pupil signal. Importantly, the recovered attentional response resembles the time course of attentional allocation previously predicted by several models of Attentional Blink. Together, these findings demonstrate that the dynamics of latent attentional episodes can be recovered from the pupil response with a temporal resolution previously obscured by the sluggishness of the pupillary response. The proposed method is also provided as a usable software package that serves as a general tool for investigating the temporal dynamics of attention in rapid visual paradigms.

Attentional selection not only increases behavioural performance and neural information for attended stimuli but, in some instances, induces changes in stimulus appearance. While reported appearance changes have mostly concerned the effects of attending to a particular spatial location, here we report a shift in appearance that occurs when attention is directed to one component of a stimulus rapidly alternating at a single spatial location. Our stimuli included two frames of different colors, which alternated at 3.75Hz or 7.5Hz. Participants reported the perceived dominance of one frame's color in the reference stimulus, by comparison with a target stimulus. We found that when one stimulus frame was more salient than the other, with a greater likelihood of capturing exogenous attention, there was an illusory perception that the frame was physically present for a greater proportion of time. When the frames were matched in salience, a task encouraging participants to endogenously direct their attention to one frame produced a smaller shift in appearance. Our results add to the body of literature demonstrating that attention can change appearance, in this case altering the appearance of attended versus unattended stimuli that are presented at the same spatial location. Furthermore, our results demonstrate that at the 'temporal transparent' alternation rate of 7.5Hz, where rapidly alternating stimuli are perceived as two translucent surfaces, attentional selection of one surface can account for the apparent paradox that feature conjunctions are perceived at stimulus alternation rates higher than predicted by the temporal limits of feature binding for other stimuli.

The retina encodes visual information through parallel pathways formed by distinct retinal ganglion (RGC) types. Current understanding of how each cell type contributes to downstream visual processing and perception is limited by a lack of tools for selectively manipulating the activities of individual RGC types. Here, we introduce a computational framework called STORM (SpatioTemporal Orthogonal Response Modulation) that creates visual stimuli that aim to selectively modulate the activities of specific RGC types while minimally affecting the activities of nearby RGCs of non-targeted types. To illustrate the approach, demonstrate its flexibility, and identify conditions under which each simulated RGC type could be selectively manipulated, we first applied this framework to simplified models of RGC function, such as Linear-Nonlinear-Poisson (LNP) and LN-LN cascade models. We also deployed this approach on more complex (and more accurate) models of RGC responses, based on convolutional neural networks. We demonstrate that differences in spatial receptive fields, temporal dynamics, response polarities, and nonlinearities can provide sufficient degrees of freedom for designing stimuli that selectively manipulate individual RGC types. We also generalize this approach to conditions where the receptive field locations are unknown for in vivo and clinical applications.

Although human perception appears continuous, converging evidence suggests that sensory input is integrated over discrete temporal windows. The duration of these Temporal Integration Windows (TIWs) determines the temporal resolution of perception, but how this fundamental property develops across infancy and childhood remains unknown. Here, we estimate TIW duration from 5 months of age to adulthood using a novel, instruction-free eye-tracking paradigm that isolates pre-attentive visual integration. Across four experiments with the same paradigm (N = 308 participants), we measured spontaneous detection of a visual target that emerges only when two alternating displays are integrated within a single perceptual window. We observed a striking developmental shortening of the TIW. In adults and 5-year-olds, integration succeeded at short interstimulus intervals (ISIs; 16-32 ms) but failed at longer intervals (83-116 ms), yielding an estimated TIW of [~]57 ms. At 3 years, the TIW extended to [~]100 ms. In infancy, integration remained robust at substantially longer ISIs, with estimates of [~]250 ms at 18 and 8 months, and [~]400 ms at 5 months. The steepest reduction occurred within the first year of life. These findings reveal an approximately sevenfold acceleration in the temporal resolution of visual perception from early infancy to adulthood. Progressive shortening of TIWs suggests that infants and adults integrate sensory information over fundamentally different temporal scales, with implications for perceptual organization, learning, and neurodevelopmental trajectories.

Age is a fundamental social attribute that affects rights, responsibilities and even social status. Color information of the face can affect perceived age, but the nature of its contribution, independent of luminance information, remains underexplored. To better characterize the role of color in age perception, we conducted two experiments. In our first experiment, we tested whether natural age-related changes in facial color alone can drive changes in perceived age. In our second experiment, we manipulated red-green contrast across faces aged 18-91 and tested the effect of this manipulation of perceived youthfulness. Our findings demonstrate that natural age-related color changes do provide cues for age perception, dissociable from luminance information. We also found that the optimal red-green contrast level to make faces look the youngest was significantly higher than the originals, meaning that faces tended to appear younger when their redness was accentuated, even though average face redness increases with age. Our analysis revealed that younger faces required greater increases in redness level compared to their older counterparts to achieve the youngest appearance. These results suggest that the spatial distribution of redness contributes to apparent youthfulness, especially in younger faces.

Unveiling the plastic mechanisms of the primary visual cortex (V1) is crucial for understanding how visual processing can change and adapt, ultimately informing interventions to promote functional recovery following visual system damage. Paired associative stimulation (PAS) protocols can induce associative plasticity within the human V1. However, previous paradigms used full-field visual stimulation, engaging both hemispheres simultaneously, leaving unclear whether plasticity can be selectively targeted to a single visual pathway and whether it can modulate the non-stimulated hemisphere. Across three experiments in healthy participants, we investigated the neurophysiological, behavioural, and interhemispheric outcomes of a novel lateralized visual PAS protocol (vPAS). Pattern-reversal checkerboards were presented in a single visual hemifield and time-locked to transcranial magnetic stimulation (TMS) pulses delivered over the contralateral V1. Experiment 1 assessed neurophysiological effects by recording hemifield-lateralized visual-evoked potentials (VEPs) and testing three inter-stimulus intervals (ISIs: 75, 100, and 125 ms) between vPAS-paired stimuli. Experiment 2 evaluated perceptual changes in contrast detection, while Experiment 3 examined vPAS hemispheric specificity and aftereffects on interhemispheric visuomotor transfer. Only the 75-ms ISI consistently modulated visual processing in the non-conditioned hemisphere, reducing N75 amplitude and improving contrast detection for visual stimuli presented contralateral to the TMS side. Longer ISIs were not effective. Furthermore, this protocol altered interhemispheric transfer exclusively when the left, but not the right, V1 was targeted during vPAS. In conclusion, lateralized vPAS induces timing-dependent changes beyond the stimulated cortex, suggesting that associative plasticity can engage contralateral visual circuits and modulate visual processing.

Efficient motion perception requires integrating immediate sensory evidence with internal representations shaped by experience. This influence of recent history can produce serial dependence effects (SDEs), in which current judgments are biased toward or away from the source of previous information. We examined SDEs in auditory, visual, and cued audiovisual motion-direction discrimination using aggregate psychometric measures and trial-level generalized linear mixed-effects models (GLMMs), which estimated the independent contributions of the previous stimulus direction and the participant's previous response. Participants completed unisensory (n=57) and cued audiovisual (n=48) motion-discrimination tasks. Aggregate analyses indicated attractive SDEs in auditory conditions, repulsive SDEs in visual conditions, and no net SDE in the audiovisual-cued condition. Trial-level modeling revealed a more nuanced structure: previous stimulus direction produced repulsive effects in both unisensory conditions, with previous visual direction influencing judgments across all conditions containing visual motion, whereas previous response produced attractive effects only when visual information was task-relevant. In the audiovisual-cued condition, the absence of a net psychometric shift was consistent with the opposing influences of an attractive response-history effect and a repulsive visual stimulus-history effect rather than an absence of serial dependence. These findings suggest that serial dependence in motion perception reflects dissociable and concurrently operating processes, including a predominantly visual, adaptation-like stimulus-history effect and a task-dependent, decision-related response-history effect.

While much of the neural and cognitive machinery for visual perception is in place within weeks from birth, we identified and quantified a fundamental difference between infant and adult vision: the temporal rate at which meaningful visual information can be reliably extracted. Using frequency-tagging electroencephalography with rapid serial visual presentation, we found that object categorization at 4 and 9 months is respectively seven times and two to three times slower than in adults. Using quantitative information from infants experiments, we created and presented accelerated visual streams to adults, to match infants perceptual experience. We used deep convolutional neural networks to identify object representations that match those elicited in adults when accelerating the information flow. Results show that acceleration compromises finer-grained object distinction while sparing broad, general object representations. We propose that slow vision is a design feature of early development for preserving and highlighting the most stable and general features of the world.

Primates use eye movements to explore and gather information about their external, visual world. Similarly, even when there is nothing to see spontaneous eye movements reflect the structure and the exploration of internal, conceptual knowledge. However, it remains unclear what mechanism links eye movements to this internal search process. One possibility is that eye movements reflect visual imagery or visual exploration strategies used to explore conceptual knowledge. On the other hand, eye movements may reflect attentional movements in the conceptual space, which are independent from visual experience. Early blind individuals are a useful testbed for this hypothesis in that they never had access to functional vision, thus ruling out the influence of imagery or recycling of visual exploration strategies. To this end, we asked early blind individuals to perform a random number generation task while recording their eye movements. Using electrooculography (EOG), we found that eye movements track the exploration of the mental number line in both sighted and blind: participants moved their eyes to the right for positive changes in magnitude in the number sequence and to the left for negative changes, with eye movement amplitude scaling with the magnitude difference. Furthermore, sighted and early blind participants showed a similar temporal profile, with the effect peaking around 750 ms before number onset. These findings support the hypothesis that internally generated shifts of attention, rather than eye movements per se, drive the exploration of conceptual spaces.

The primate retina is strikingly non-uniform, with receptor density falling off sharply from fovea to periphery. Every eye-movement brings a previously low-acuity peripheral area to high-acuity foveal processing. We hypothesize that this process provides a natural self-supervision signal, enabling representation learning by predicting fine details in the periphery. We implement this in a SimMIM-style vision transformer, replacing its blank masks (ViT-Blank) with a uniform blur mask (ViT-Blur) and further with retinal masks in which a foveal patch is randomly selected, and all other patches are progressively blurred based on eccentricity from the foveal patch (ViT-Retina). ViTs were pre-trained to reconstruct the full-resolution image from the corrupted input and then fine-tuned on clean images for classification. ViT-Blur and ViT-Retina outperform ViT-Blank on the pre-training reconstruction task while controlling for retained information from the input across corruption types. Critically, they also outperform ViT-Blank on downstream classification, showing that they learn representations that generalize better. The ViT-Blur and ViT-Retina are also more robust in impoverishment conditions such as higher mask ratios, fewer pre-training epochs, smaller training sets, and test time image corruptions. Using additive band-limited noise, we discovered that ViT-Blank relies on higher spatial frequencies compared to ViT-Blur and ViT-Retina, and we show this to be the likely cause of poorer generalization: the centroid of a model's spatial frequency tuning is negatively correlated with its classification accuracy ($r = -0.73$). We further show that a closed-form linear-ridge encoder derived from the same reconstruction objective reproduces this tuning ($r = 0.88$). Overall, our experiments show the promise of a biologically-inspired self-supervision objective for learning robust visual representations.

Recognising people precedes and informs most social interactions, yet it is not clear why individuals differ so markedly in face recognition ability. Exposure-based accounts propose that it is shaped by visual experience with faces, known as 'face diet', but most prior work treats exposure as passively acquired, thereby masking the active and dynamic nature of visual attention in natural settings. Here we examined whether spontaneous attention to faces in everyday environments predicts face recognition ability. Eighty participants wore mobile eye-tracking glasses while walking through a busy university campus and during a face-to-face interaction, before completing standard measures of face recognition ability. Fixations directed to faces showed stable individual differences in both settings. People who looked more at faces scored higher on our composite measure of face recognition ability, and gaze in the two settings predicted ability independently, together accounting for 22% of its variance. These findings suggest that differences in how people actively sample visual information from their environments may be an overlooked source of variance in perceptual expertise.

As animals explore their environment, self-generated motion can affect visual perception, while perceptual constraints shape movement decisions. Movement strategies should therefore balance the benefits of exploration against the potential perceptual costs of motion. Here, we test whether the movement strategies animals adopt can enhance information acquisition while accounting for the constraints that motion imposes on visual perception. We first quantified the visual perceptual abilities of freely moving fish in three dimensions by projecting virtual prey around three-spined sticklebacks (Gasterosteus aculeatus), determining how the likelihood of detecting prey varied across the fish's visual field as a function of their movement. Self-induced motion reduced the likelihood prey were detected, with increases in speed non-linearly reducing the perceptual abilities of the fish. Using agent-based simulations based on empirical models of visual perception, we demonstrate that when visual perception degrades non-linearly with speed, the strategy that maximizes prey detection is 'saltatory' locomotion, involving stationary pauses interspersed with movements. Moreover, experiments with freely swimming fish showed they adopted specific durations of stops and movements that improved prey detection given the way motion impacts their perception. Our results reveal that some movement strategies are more effective than others at gathering information during exploration.

What we see at one location depends on what surrounds it. Such contextual e[ff]ects are thought to arise from neural signals that travel across the retinotopic representations.Because these signals need more time to travel longer distances, a more distant surround should influence perception most strongly when presented earlier, compensating for its delay. We tested this prediction in human observers using a visual illusion in which a central target appears tilted away from the orientation of its surround (tilt illusion). Measuring the temporal window of this illusion using psychophysical reverse correlation revealed the opposite: the most e[ff]ective surround occurred progressively later as distance increased. Computational modelling demonstrated that this reversal is explained if the visual system sends out a query that recruits contextual information on demand. Perceptual appearance may thus be constructed through temporally unfolding interactions in which context is actively gathered, not passively received.

Revising a decision while acting on it is a fundamental aspect of natural behavior, yet the underlying computations are difficult to track in real time. To explore how such belief updates unfold, we tested whether eye movements reveal a subject's change of mind during a naturalistic navigation task with a binary target choice. Humans and monkeys were required to navigate to one of two goal locations presented either simultaneously or sequentially in a virtual environment where explicit position cues were absent. Our findings reveal that gaze continuously tracks the latent variable, including changes of mind. We found that the target tracking accuracy before the second target is even presented during sequential target trials predicts whether the subject's navigational goal persists (first target choice) or changes (second target choice). These results underscore the importance of gaze in action planning and changes of mind, supporting the hypothesis that eye movements embody not only the current latent variable but also its uncertainty.

Despite the ubiquitous nature of visual search in our daily lives, our knowledge of the underlying mechanisms is almost entirely derived from studies using simple, synthetic stimuli briefly flashed on a computer monitor. In real life, however, visual search occurs in three-dimensional environments over much longer timescales and involves the dynamic coordination of attention, memory, and action planning, generating novel predictions that cannot be tested with traditional approaches. Here, we examined how self-guided visual search unfolds under naturalistic conditions that extend over space and time. We asked participants (N=80) to find specific objects (e.g., a green lamp, an orange shelf) in an immersive, visually rich virtual reality environment using range-limited teleportation (2 m steps). Consistent with our hypotheses, we found that self-guided search engaged a hierarchical information-seeking policy characterized at each stage by recurring plan-and-execute cycles paired with subgoal-specific attentional priorities. Subgoal priorities progressed from wayfinding to identification of clusters of target-similar objects, and then to specific target features. These search priorities progressively narrowed the search space through coordinated eye, head, and body movements that recurred throughout the task. We conclude that naturalistic search is hierarchical, built from sequential subgoals with shifting attentional priorities that single-level tasks cannot capture. Moreover, immersive VR with eye-head-body tracking dissociates motor planning from execution moments and reveals how looking content shifts with each task subgoal while the person moves through space, something desktop tasks cannot capture.

When we appreciate visual art, we often take our time: We visit museums or galleries and deliberately engage with artworks and their meaning. Yet, our liking might at least partly be shaped by incidental encoding of the artworks' visual features, a process initially independent from sustained engagement. Here, we show that liking for visual art can be predicted from rapid and automatic responses in the visual brain. We recorded EEG responses while participants viewed 999 artworks from the Vienna Art Picture System (VAPS), which features complex artworks that span different categories and art styles. Artworks were shown with a 300ms stimulus-onset-asynchrony (150ms on, 150 off), and participants responded to occasional target images. Using representational similarity analysis, we predicted the geometry of neural representations from the artworks' similarity in liking (based on ratings from a different group of observers), while simultaneously controlling for similarities in other properties (category, art style, valence, arousal, and familiarity). Crucially, neural responses emerging within the first 150ms were robustly connected to liking, even though the task did not afford any liking judgments. Strikingly, this early neural correlate of liking was also observed in a second experiment, in which we presented the same 999 artworks with a 50ms stimulus-onset-asynchrony. Our results show that even under ultra-fast presentation conditions, automatic neural responses rapidly indicate liking. While fully appreciating an artwork may still need deliberate and prolonged engagement, the perceptual basis of whether we like or dislike an artwork seems to be laid in a split second.